<!DOCTYPE html>

<html>
  <head>
    <meta charset="utf-8">
    
    <title>Parallel Random Number Generation &mdash; NumPy v1.18 Manual</title>
    
    <link rel="stylesheet" type="text/css" href="../../_static/css/spc-bootstrap.css">
    <link rel="stylesheet" type="text/css" href="../../_static/css/spc-extend.css">
    <link rel="stylesheet" href="../../_static/scipy.css" type="text/css" >
    <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" >
    <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" >
    
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '../../',
        VERSION:     '1.18.1',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  false
      };
    </script>
    <script type="text/javascript" src="../../_static/jquery.js"></script>
    <script type="text/javascript" src="../../_static/underscore.js"></script>
    <script type="text/javascript" src="../../_static/doctools.js"></script>
    <script type="text/javascript" src="../../_static/language_data.js"></script>
    <script type="text/javascript" src="../../_static/js/copybutton.js"></script>
    <link rel="author" title="About these documents" href="../../about.html" >
    <link rel="index" title="Index" href="../../genindex.html" >
    <link rel="search" title="Search" href="../../search.html" >
    <link rel="top" title="NumPy v1.18 Manual" href="../../index.html" >
    <link rel="up" title="Random sampling (numpy.random)" href="index.html" >
    <link rel="next" title="Multithreaded Generation" href="multithreading.html" >
    <link rel="prev" title="numpy.random.SeedSequence.spawn" href="bit_generators/generated/numpy.random.SeedSequence.spawn.html" > 
  </head>
  <body>
<div class="container">
  <div class="top-scipy-org-logo-header" style="background-color: #a2bae8;">
    <a href="../../index.html">
      <img border=0 alt="NumPy" src="../../_static/numpy_logo.png"></a>
    </div>
  </div>
</div>


    <div class="container">
      <div class="main">
        
	<div class="row-fluid">
	  <div class="span12">
	    <div class="spc-navbar">
              
    <ul class="nav nav-pills pull-left">
        <li class="active"><a href="https://numpy.org/">NumPy.org</a></li>
        <li class="active"><a href="https://numpy.org/doc">Docs</a></li>
        
        <li class="active"><a href="../../index.html">NumPy v1.18 Manual</a></li>
        

          <li class="active"><a href="../index.html" >NumPy Reference</a></li>
          <li class="active"><a href="../routines.html" >Routines</a></li>
          <li class="active"><a href="index.html" accesskey="U">Random sampling (<code class="xref py py-mod docutils literal notranslate"><span class="pre">numpy.random</span></code>)</a></li> 
    </ul>
              
              
    <ul class="nav nav-pills pull-right">
      <li class="active">
        <a href="../../genindex.html" title="General Index"
           accesskey="I">index</a>
      </li>
      <li class="active">
        <a href="multithreading.html" title="Multithreaded Generation"
           accesskey="N">next</a>
      </li>
      <li class="active">
        <a href="bit_generators/generated/numpy.random.SeedSequence.spawn.html" title="numpy.random.SeedSequence.spawn"
           accesskey="P">previous</a>
      </li>
    </ul>
              
	    </div>
	  </div>
	</div>
        

	<div class="row-fluid">
      <div class="spc-rightsidebar span3">
        <div class="sphinxsidebarwrapper">
  <h3><a href="../../contents.html">Table of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">Parallel Random Number Generation</a><ul>
<li><a class="reference internal" href="#seedsequence-spawning"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code> spawning</a></li>
<li><a class="reference internal" href="#independent-streams">Independent Streams</a></li>
<li><a class="reference internal" href="#jumping-the-bitgenerator-state">Jumping the BitGenerator state</a></li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="bit_generators/generated/numpy.random.SeedSequence.spawn.html"
                        title="previous chapter">numpy.random.SeedSequence.spawn</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="multithreading.html"
                        title="next chapter">Multithreaded Generation</a></p>
<div id="searchbox" style="display: none" role="search">
  <h4>Quick search</h4>
    <div>
    <form class="search" action="../../search.html" method="get">
      <input type="text" style="width: inherit;" name="q" />
      <input type="submit" value="search" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    </div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
          <div class="span9">
            
        <div class="bodywrapper">
          <div class="body" id="spc-section-body">
            
  <div class="section" id="parallel-random-number-generation">
<h1>Parallel Random Number Generation<a class="headerlink" href="#parallel-random-number-generation" title="Permalink to this headline">¶</a></h1>
<p>There are three strategies implemented that can be used to produce
repeatable pseudo-random numbers across multiple processes (local
or distributed).</p>
<div class="section" id="seedsequence-spawning">
<span id="seedsequence-spawn"></span><h2><a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a> spawning<a class="headerlink" href="#seedsequence-spawning" title="Permalink to this headline">¶</a></h2>
<p><a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a> <a class="reference external" href="http://www.pcg-random.org/posts/developing-a-seed_seq-alternative.html">implements an algorithm</a> to process a user-provided seed,
typically as an integer of some size, and to convert it into an initial state for
a <a class="reference internal" href="bit_generators/generated/numpy.random.BitGenerator.html#numpy.random.BitGenerator" title="numpy.random.BitGenerator"><code class="xref py py-obj docutils literal notranslate"><span class="pre">BitGenerator</span></code></a>. It uses hashing techniques to ensure that low-quality seeds
are turned into high quality initial states (at least, with very high
probability).</p>
<p>For example, <a class="reference internal" href="bit_generators/mt19937.html#numpy.random.MT19937" title="numpy.random.MT19937"><code class="xref py py-obj docutils literal notranslate"><span class="pre">MT19937</span></code></a> has a state consisting of 624
<em class="xref py py-obj">uint32</em> integers. A naive way to take a 32-bit integer seed would be to just set
the last element of the state to the 32-bit seed and leave the rest 0s. This is
a valid state for <a class="reference internal" href="bit_generators/mt19937.html#numpy.random.MT19937" title="numpy.random.MT19937"><code class="xref py py-obj docutils literal notranslate"><span class="pre">MT19937</span></code></a>, but not a good one. The Mersenne Twister
algorithm <a class="reference external" href="http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html">suffers if there are too many 0s</a>. Similarly, two adjacent 32-bit
integer seeds (i.e. <code class="docutils literal notranslate"><span class="pre">12345</span></code> and <code class="docutils literal notranslate"><span class="pre">12346</span></code>) would produce very similar
streams.</p>
<p><a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a> avoids these problems by using successions of integer hashes
with good <a class="reference external" href="https://en.wikipedia.org/wiki/Avalanche_effect">avalanche properties</a> to ensure that flipping any bit in the input
input has about a 50% chance of flipping any bit in the output. Two input seeds
that are very close to each other will produce initial states that are very far
from each other (with very high probability). It is also constructed in such
a way that you can provide arbitrary-sized integers or lists of integers.
<a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a> will take all of the bits that you provide and mix them
together to produce however many bits the consuming <a class="reference internal" href="bit_generators/generated/numpy.random.BitGenerator.html#numpy.random.BitGenerator" title="numpy.random.BitGenerator"><code class="xref py py-obj docutils literal notranslate"><span class="pre">BitGenerator</span></code></a> needs to
initialize itself.</p>
<p>These properties together mean that we can safely mix together the usual
user-provided seed with simple incrementing counters to get <a class="reference internal" href="bit_generators/generated/numpy.random.BitGenerator.html#numpy.random.BitGenerator" title="numpy.random.BitGenerator"><code class="xref py py-obj docutils literal notranslate"><span class="pre">BitGenerator</span></code></a>
states that are (to very high probability) independent of each other. We can
wrap this together into an API that is easy to use and difficult to misuse.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">numpy.random</span> <span class="kn">import</span> <span class="n">SeedSequence</span><span class="p">,</span> <span class="n">default_rng</span>

<span class="n">ss</span> <span class="o">=</span> <span class="n">SeedSequence</span><span class="p">(</span><span class="mi">12345</span><span class="p">)</span>

<span class="c1"># Spawn off 10 child SeedSequences to pass to child processes.</span>
<span class="n">child_seeds</span> <span class="o">=</span> <span class="n">ss</span><span class="o">.</span><span class="n">spawn</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="n">streams</span> <span class="o">=</span> <span class="p">[</span><span class="n">default_rng</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">child_seeds</span><span class="p">]</span>
</pre></div>
</div>
<p>Child <a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a> objects can also spawn to make grandchildren, and so on.
Each <a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a> has its position in the tree of spawned <a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a>
objects mixed in with the user-provided seed to generate independent (with very
high probability) streams.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">grandchildren</span> <span class="o">=</span> <span class="n">child_seeds</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">spawn</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="n">grand_streams</span> <span class="o">=</span> <span class="p">[</span><span class="n">default_rng</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">grandchildren</span><span class="p">]</span>
</pre></div>
</div>
<p>This feature lets you make local decisions about when and how to split up
streams without coordination between processes. You do not have to preallocate
space to avoid overlapping or request streams from a common global service. This
general “tree-hashing” scheme is <a class="reference external" href="https://www.iro.umontreal.ca/~lecuyer/myftp/papers/parallel-rng-imacs.pdf">not unique to numpy</a> but not yet widespread.
Python has increasingly-flexible mechanisms for parallelization available, and
this scheme fits in very well with that kind of use.</p>
<p>Using this scheme, an upper bound on the probability of a collision can be
estimated if one knows the number of streams that you derive. <a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a>
hashes its inputs, both the seed and the spawn-tree-path, down to a 128-bit
pool by default. The probability that there is a collision in
that pool, pessimistically-estimated (<a class="footnote-reference brackets" href="#id3" id="id1">1</a>), will be about <img class="math" src="../../_images/math/8442c9ebbbd397960513b77162319907436195ae.svg" alt="n^2*2^{-128}"/> where
<em class="xref py py-obj">n</em> is the number of streams spawned. If a program uses an aggressive million
streams, about <img class="math" src="../../_images/math/aff3a1944abee3924eeb3054dfb155502a7583e2.svg" alt="2^{20}"/>, then the probability that at least one pair of
them are identical is about <img class="math" src="../../_images/math/5316dc12ee9d89833244f0b38e81954055cab1af.svg" alt="2^{-88}"/>, which is in solidly-ignorable
territory (<a class="footnote-reference brackets" href="#id4" id="id2">2</a>).</p>
<dl class="footnote brackets">
<dt class="label" id="id3"><span class="brackets"><a class="fn-backref" href="#id1">1</a></span></dt>
<dd><p>The algorithm is carefully designed to eliminate a number of possible
ways to collide. For example, if one only does one level of spawning, it
is guaranteed that all states will be unique. But it’s easier to
estimate the naive upper bound on a napkin and take comfort knowing
that the probability is actually lower.</p>
</dd>
<dt class="label" id="id4"><span class="brackets"><a class="fn-backref" href="#id2">2</a></span></dt>
<dd><p>In this calculation, we can ignore the amount of numbers drawn from each
stream. Each of the PRNGs we provide has some extra protection built in
that avoids overlaps if the <a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a> pools differ in the
slightest bit. <a class="reference internal" href="bit_generators/pcg64.html#numpy.random.PCG64" title="numpy.random.PCG64"><code class="xref py py-obj docutils literal notranslate"><span class="pre">PCG64</span></code></a> has <img class="math" src="../../_images/math/eff8c0727490d92520743ec0f4f745db2b9a8133.svg" alt="2^{127}"/> separate cycles
determined by the seed in addition to the position in the
<img class="math" src="../../_images/math/91018ad7005c40f683c645b99dd28ae741d2de00.svg" alt="2^{128}"/> long period for each cycle, so one has to both get on or
near the same cycle <em>and</em> seed a nearby position in the cycle.
<a class="reference internal" href="bit_generators/philox.html#numpy.random.Philox" title="numpy.random.Philox"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Philox</span></code></a> has completely independent cycles determined by the seed.
<a class="reference internal" href="bit_generators/sfc64.html#numpy.random.SFC64" title="numpy.random.SFC64"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SFC64</span></code></a> incorporates a 64-bit counter so every unique seed is at
least <img class="math" src="../../_images/math/47e87a1638fac5fa3416ef91f07e6f3572eed8a6.svg" alt="2^{64}"/> iterations away from any other seed. And
finally, <a class="reference internal" href="bit_generators/mt19937.html#numpy.random.MT19937" title="numpy.random.MT19937"><code class="xref py py-obj docutils literal notranslate"><span class="pre">MT19937</span></code></a> has just an unimaginably huge period. Getting
a collision internal to <a class="reference internal" href="bit_generators/generated/numpy.random.SeedSequence.html#numpy.random.SeedSequence" title="numpy.random.SeedSequence"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SeedSequence</span></code></a> is the way a failure would be
observed.</p>
</dd>
</dl>
</div>
<div class="section" id="independent-streams">
<span id="id5"></span><h2>Independent Streams<a class="headerlink" href="#independent-streams" title="Permalink to this headline">¶</a></h2>
<p><a class="reference internal" href="bit_generators/philox.html#numpy.random.Philox" title="numpy.random.Philox"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Philox</span></code></a> is a counter-based RNG based which generates values by
encrypting an incrementing counter using weak cryptographic primitives. The
seed determines the key that is used for the encryption. Unique keys create
unique, independent streams. <a class="reference internal" href="bit_generators/philox.html#numpy.random.Philox" title="numpy.random.Philox"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Philox</span></code></a> lets you bypass the
seeding algorithm to directly set the 128-bit key. Similar, but different, keys
will still create independent streams.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">secrets</span>
<span class="kn">from</span> <span class="nn">numpy.random</span> <span class="kn">import</span> <span class="n">Philox</span>

<span class="c1"># 128-bit number as a seed</span>
<span class="n">root_seed</span> <span class="o">=</span> <span class="n">secrets</span><span class="o">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="mi">128</span><span class="p">)</span>
<span class="n">streams</span> <span class="o">=</span> <span class="p">[</span><span class="n">Philox</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">root_seed</span> <span class="o">+</span> <span class="n">stream_id</span><span class="p">)</span> <span class="k">for</span> <span class="n">stream_id</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)]</span>
</pre></div>
</div>
<p>This scheme does require that you avoid reusing stream IDs. This may require
coordination between the parallel processes.</p>
</div>
<div class="section" id="jumping-the-bitgenerator-state">
<span id="parallel-jumped"></span><h2>Jumping the BitGenerator state<a class="headerlink" href="#jumping-the-bitgenerator-state" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">jumped</span></code> advances the state of the BitGenerator <em>as-if</em> a large number of
random numbers have been drawn, and returns a new instance with this state.
The specific number of draws varies by BitGenerator, and ranges from
<img class="math" src="../../_images/math/47e87a1638fac5fa3416ef91f07e6f3572eed8a6.svg" alt="2^{64}"/> to <img class="math" src="../../_images/math/91018ad7005c40f683c645b99dd28ae741d2de00.svg" alt="2^{128}"/>.  Additionally, the <em>as-if</em> draws also depend
on the size of the default random number produced by the specific BitGenerator.
The BitGenerators that support <code class="docutils literal notranslate"><span class="pre">jumped</span></code>, along with the period of the
BitGenerator, the size of the jump and the bits in the default unsigned random
are listed below.</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 18%" />
<col style="width: 27%" />
<col style="width: 27%" />
<col style="width: 27%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>BitGenerator</p></th>
<th class="head"><p>Period</p></th>
<th class="head"><p>Jump Size</p></th>
<th class="head"><p>Bits</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>MT19937</p></td>
<td><p><img class="math" src="../../_images/math/f43c200a729a36c39ded3a221875903a756195e7.svg" alt="2^{19937}"/></p></td>
<td><p><img class="math" src="../../_images/math/91018ad7005c40f683c645b99dd28ae741d2de00.svg" alt="2^{128}"/></p></td>
<td><p>32</p></td>
</tr>
<tr class="row-odd"><td><p>PCG64</p></td>
<td><p><img class="math" src="../../_images/math/91018ad7005c40f683c645b99dd28ae741d2de00.svg" alt="2^{128}"/></p></td>
<td><p><img class="math" src="../../_images/math/2476ed1fcb927c7dfbafa9ee1a4c5d749fdb3ff5.svg" alt="~2^{127}"/> (<a class="footnote-reference brackets" href="#id7" id="id6">3</a>)</p></td>
<td><p>64</p></td>
</tr>
<tr class="row-even"><td><p>Philox</p></td>
<td><p><img class="math" src="../../_images/math/0a4fc3833afca88a800dfb98cda1716a112f29fb.svg" alt="2^{256}"/></p></td>
<td><p><img class="math" src="../../_images/math/91018ad7005c40f683c645b99dd28ae741d2de00.svg" alt="2^{128}"/></p></td>
<td><p>64</p></td>
</tr>
</tbody>
</table>
<dl class="footnote brackets">
<dt class="label" id="id7"><span class="brackets"><a class="fn-backref" href="#id6">3</a></span></dt>
<dd><p>The jump size is <img class="math" src="../../_images/math/70153266a65ec5b81cf96567a3665e6f869a1c9c.svg" alt="(\phi-1)*2^{128}"/> where <img class="math" src="../../_images/math/fffd2357ee88a9c50ba9e831ed64c39c73d54a07.svg" alt="\phi"/> is the
golden ratio. As the jumps wrap around the period, the actual distances
between neighboring streams will slowly grow smaller than the jump size,
but using the golden ratio this way is a classic method of constructing
a low-discrepancy sequence that spreads out the states around the period
optimally. You will not be able to jump enough to make those distances
small enough to overlap in your lifetime.</p>
</dd>
</dl>
<p><code class="docutils literal notranslate"><span class="pre">jumped</span></code> can be used to produce long blocks which should be long enough to not
overlap.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">secrets</span>
<span class="kn">from</span> <span class="nn">numpy.random</span> <span class="kn">import</span> <span class="n">PCG64</span>

<span class="n">seed</span> <span class="o">=</span> <span class="n">secrets</span><span class="o">.</span><span class="n">getrandbits</span><span class="p">(</span><span class="mi">128</span><span class="p">)</span>
<span class="n">blocked_rng</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">rng</span> <span class="o">=</span> <span class="n">PCG64</span><span class="p">(</span><span class="n">seed</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
    <span class="n">blocked_rng</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">rng</span><span class="o">.</span><span class="n">jumped</span><span class="p">(</span><span class="n">i</span><span class="p">))</span>
</pre></div>
</div>
<p>When using <code class="docutils literal notranslate"><span class="pre">jumped</span></code>, one does have to take care not to jump to a stream that
was already used. In the above example, one could not later use
<code class="docutils literal notranslate"><span class="pre">blocked_rng[0].jumped()</span></code> as it would overlap with <code class="docutils literal notranslate"><span class="pre">blocked_rng[1]</span></code>. Like
with the independent streams, if the main process here wants to split off 10
more streams by jumping, then it needs to start with <code class="docutils literal notranslate"><span class="pre">range(10,</span> <span class="pre">20)</span></code>,
otherwise it would recreate the same streams. On the other hand, if you
carefully construct the streams, then you are guaranteed to have streams that
do not overlap.</p>
</div>
</div>


          </div>
        </div>
          </div>
        </div>
      </div>
    </div>

    <div class="container container-navbar-bottom">
      <div class="spc-navbar">
        
      </div>
    </div>
    <div class="container">
    <div class="footer">
    <div class="row-fluid">
    <ul class="inline pull-left">
      <li>
        &copy; Copyright 2008-2019, The SciPy community.
      </li>
      <li>
      Last updated on Feb 20, 2020.
      </li>
      <li>
      Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 2.4.2.
      </li>
    </ul>
    </div>
    </div>
    </div>
  </body>
</html>